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Abstract: 

The  standard  for  Product  Data  Representation  and  Exchange,  ISO  10303,  defines  a neutral, 
computer-interpretable  representation  for  describing  product  data  in  a manner  that  is 
independent  from  any  particular  system.  This  standard  is  more  commonly  known  as  STEP, 
STandard  for  the  Exchange  of  Product  model  data  and  it  is  designed  to  support  a wide  range 
of  design,  engineering,  and  product  support  applications.  Support  for  specific  applications  is 
provided  through  an  Application  Protocol. 

This  document  describes  the  policy  and  procedures  for  a "beta"  conformance  testing 
program  for  one  of  these  application  protocols:  Configuration  Controlled  3D  Design  for 
Mechanical  Parts  and  Assemblies  (ISO  10303-203).  However,  the  concepts  described  are 
applicable  to  ISO  10303  in  general  and  the  characteristics  of  the  longer  term  testing  program 
are  described. 

The  objective  of  conformance  testing  is  to  maximize  the  probability  of  successful  exchange 
among  systems  which  implement  a standard.  This  document  describes  the  policy  and 
procedures  used  by  the  National  Institute  of  Standards  and  Technology  (NIST)  and  the 
Industrial  Technology  Institute  (ITI)  for  the  Beta  STEP  Conformance  Testing  Program.  The 
objective  of  this  beta  program  is  to  develop  an  acceptable  degree  of  confidence  in  the  test 
methods  and  procedures.  Attaining  this  confidence  will  be  accomplished  by  trial  use  of  the 
test  methods  and  procedures  with  developers  of  early  implementations  of  ISO  10303-203. 
This  testing  program  is  being  conducted  under  a cooperative  agreement  between  NIST’s 
Manufacturing  Engineering  Laboratory  and  the  ITI’s  Center  for  Electronic  Commerce. 
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1.  Introduction 


The  National  Institute  of  Standards  and  Technology  (NIST)  works  through  voluntary  industry 
standards  organizations  to  develop  standards  that  meet  the  needs  of  industry  and  government 
users.  To  be  viable,  it  is  important  that  these  standards  meet  a real  industrial  need.  One 
such  standard  is  ISO  10303,  Product  Data  Representation  and  Exchange,  which  is  more 
commonly  referred  to  as  STEP,  STandard  for  the  Exchange  of  Product  model  data.  This 
standard  specifies  functional  and  data  requirements  for  product  data  exchange. 

To  effectively  use  commercial  off-the-shelf  products  which  implement  this  standard  and  to 
achieve  interoperability  among  these  systems,  conformance  testing  is  required  to  reduce  the 
risks  and  raise  the  consumer  confidence  in  these  systems.  This  document  is  specifically 
directed  at  developers  of  commercial  off-the-shelf  products  which  implement  STEP  and 
testing  laboratories,  but  users  of  such  systems  may  also  be  interested  in  the  planned  STEP 
conformance  testing  program. 

The  objectives  of  the  NIST  STEP  Conformance  Policy  are: 

• To  reduce  the  total  information  systems  costs  by  making  it  more  predictable  and  less 
expensive  to  maintain  software  applications  and  to  transfer  compatible  information  among 
applications  using  different  information  systems,  including  replacement  systems. 

• To  protect  the  capital  investment  of  users  of  the  standard  by  ensuring  to  the  extent 
possible  that  commercial  off-the-shelf  products  meet  the  requirements  in  the  standard, 
thereby  providing  a known  capability  which  can  be  expected  of  any  certified  product. 

• To  identify  competent  testing  laboratories  to  assist  users  of  the  standard  in  the 
procurement  of  commercial  off-the-shelf  products. 

The  STEP  conformance  testing  program  is  being  designed  to  be  as  barrier-free  as  possible 
for  suppliers  of  commercial  off-the-shelf  products.  Some  of  the  barriers  to  suppliers  which 
must  be  guarded  against  include  high  cost  (e.g.  testing  fees,  time  required,  and  logistic 
arrangements),  lack  of  useful  test  results,  lack  of  acceptance  of  test  results  internationally, 
and  lack  of  awareness  by  users  of  the  benefits  of  conformance  testing.  Self-testing  by 
vendors  duri  ^g  their  product  development  cycle  is  encouraged  and  an  information  kit  and 
tools  are  available  for  this  purpose.  For  further  information  relating  to  this  new  paradigm 
for  software  conformance  assessment,  refer  to  Issues,  Requirements,  and  Recommendations 
for  a STEP  Conformance  Testing  Program} 

The  beta  testing  program  seeks  collaborations  with  U.S.  developers  of  STEP-based  software 
products  to  ensure  delivery  of  value-added  testing  services.  Strong  vendor  involvement  is 


^ See  Appendix  A for  a list  of  document  sources. 
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needed  to  obtain  feedback  on  all  aspects  of  the  proposed  test  program.  The  objective  of  the 
STEP  beta  testing  program  is  to  accelerate  the  introduction  of  high-quality  commercial  off- 
the-shelf  products  which  meet  the  industrial  needs  specified  in  STEP. 

This  document  describes  requirements  for  the  conformance  assessment  of  STEP-based 
software  systems.  This  document  specifically  lays  out  the  requirements  for  the  conformance 
testing  of  implementations  of  the  first  standardized  STEP  Application  Protocol  (AP): 
Configuration  Controlled  3D  Design  for  Mechanical  Parts  and  Assemblies,  commonly 
referred  to  as  AP  203. 

The  objective  of  any  conformance  testing  program  is  to  maximize  the  probability  of 
successful  exchange  among  systems  which  implement  a standard.  Conformance  testing  does 
not  ensure  interoperability;  it  only  evaluates  a system  to  determine  if  it  faithfully  implements 
the  requirements  specified  by  a standard.  It  improves  the  probability  that  systems  will 
interoperate  and  helps  to  isolate  the  source  of  problems.  Any  conformance  testing  program 
must  also  ensure  comparability  of  test  results  (i.e.  testing  of  the  same  system  at  different 
sites  and  testing  laboratories  produces  the  same  test  results). 

This  document  describes  a beta  testing  program  for  STEP  and  also  describes  the  planned 
testing  policy  for  a STEP  conformance  testing  program.  The  objective  of  a beta  testing 
program  is  to  achieve  an  acceptable  degree  of  confidence  in: 

• how  conformance  testing  is  performed  which  includes  the  test  methods  and 
test  procedures; 

• what  is  used  in  the  testing  which  includes  the  test  suite  and  testing  system 
software;  and 

• what  is  being  produced  to  provide  repeatable  results  and  meaningful  test 
reports. 

The  National  Institute  of  Standards  and  Technology’s  (NIST)  National  PDES  Testbed  within 
the  Manufacturing  Engineering  Laboratory(MEL)  and  the  Industrial  Technology  Institute’s 
(ITI)  Center  for  Electronic  Commerce  (CEC)  are  conducting  the  beta  conformance  testing 
program  for  AP  203.  The  beta  testing  program  is  being  executed  to  establish  a known 
baseline  of  testing  capability.  The  beta  testing  program  is  being  performed  using  ISO/DIS 
10303-203  as  the  reference  standard.  The  international  standard  ISO  10303  has  been 
approved  and  the  publication  should  be  available  by  December  1994.  This  test  program  will 
transition  to  the  international  standard  version  after  some  reasonable  period  of  time  to  allow 
for  implementations  to  be  developed. 

This  document  is  organized  in  the  following  manner.  Section  Two  of  this  document  provides 
many  of  the  testing  terms  that  are  used  within  ISO  10303.  Section  Three  provides  additional 
background  information  and  identifies  the  reference  standards  which  are  the  basis  for  testing. 
Section  Four  states  policies  relating  to  both  the  beta  testing  program  and  the  STEP  testing 
program.  Section  Five  provides  specific  information  about  the  beta  testing  program.  Finally, 
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section  Six  describes  the  procedures  and  reporting  requirements  for  the  beta  testing  program. 
Appendix  A provides  a list  of  references  and  identifies  where  any  of  the  documents 
referenced  can  be  obtained. 

2.  Terms  and  Abbreviations 

The  following  terms  used  within  this  document  are  defined  in  ISO  10303-1,  Overview  and 
fundamental  principles,  or  ISO  10303-31,  Conformance  testing  methodology  and  framework: 
General  concepts: 

abstract  test  case  (ATC):  a specification,  encapsulating  at  least  one  test  purpose,  that 
provides  the  formal  basis  for  deriving  an  executable  test. 

abstract  test  suite  (ATS):  the  complete  set  of  abstract  test  cases  necessary  to  perform 
conformance  testing  of  an  implementation  against  a standard  or  group  of  standards. 

(laboratory)  accreditation:  the  process  of  ensuring  a testing  laboratory  is  competent  to 
carry  out  specific  (types  of)  tests.  The  term  covers  the  recognition  of  both  the  technical 
competence  and  the  impartiality  of  a testing  laboratory.^ 

Application  Protocol  (AP):  a part  of  ISO  10303  that  specifies  the  data  structures  and 
semantics  which  satisfy  the  scope  and  information  requirements  of  a specific  application. 

assertion:  A logical  expression  specifying  a set  of  conditions  that  program  variables  must 
satisfy  or  a program  state  that  must  exist  at  a particular  point  during  program  execution. 

certificate  of  conformance:  a document  issued  by  an  impartial  body  indicating  that  a 
product  meets  the  requirements  of  a specific  standard  as  determined  through  use  of  an 
approved  test  method. 

certification  authority:  an  administrative  activity  which  establishes  conformance  testing 
program  policies  and  procedures  and  which  issues  certificates  of  conformity  for  products. 

conformance  assessment  process:  the  process  of  accomplishing  the  conformance  testing 
activities  needed  to  determine  the  conformance  of  an  implementation  to  an  Application 
Protocol. 

conformance  class:  a subset  of  an  Application  Protocol  for  which  conformance  may  be 
claimed. 


^ Accreditation  is  normally  awarded  following  successful  laboratory  assessment  and  this 
is  followed  by  appropriate  surveillance. 
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conformance  requirement:  a precise  definition  of  a characteristic  required  to  be  present  in  a 
conforming  implementation. 

conformance  log:  a record  of  information,  produced  as  a result  of  a test  campaign,  that  is 
sufficient  to  make  and  verify  the  assignment  of  test  verdicts. 

conformance  testing:  the  testing  of  a product  for  the  existence  of  specific  characteristics 
required  by  a standard  in  order  to  determine  the  extent  to  which  that  product  is  a conforming 
implementation. 

data  exchange:  the  storing,  accessing,  transferring,  and  archiving  of  data. 

exchange  structure:  a computer-interpretable  format  used  for  storing,  accessing, 
transferring,  and  archiving  data. 

implementation  method:  a representation  technique  used  by  computer  systems  to  exchange 
product  data  as  described  with  the  EXPRESS  data  specification  language  (ISO  10303-11). 

fail  (verdict):  a test  verdict  given  when  the  observed  test  outcome  demonstrates 
non-conformance  with  respect  to  either  the  test  purpose  or  at  least  one  of  the  conformance 
requirements  in  the  relevant  standard(s). 

Implementation  Under  Test  (lUT):  the  product  which  is  to  be  evaluated  by  testing  and 
which  implements  the  characteristics  of  the  standard(s). 

inconclusive  (verdict):  a test  verdict  given  when  the  observed  test  outcome  is  such  that 
neither  a pass  nor  a fail  verdict  can  be  given. 

pass  (verdict):  a test  verdict  given  when  the  observed  test  outcome  gives  evidence  of 
conformance  to  the  conformance  requirement  on  which  the  test  purpose  is  focused  and  is 
valid  with  respect  to  the  relevant  standard(s)  and  with  respect  to  the  PICS. 

PICS  proforma:  a standardized  document  in  the  form  of  a questionnaire,  which,  when 
completed  for  a particular  implementation,  becomes  the  protocol  implementation 
conformance  statement. 

protocol  implementation  conformance  statement  (PICS):  a statement  of  which  capabilities 
and  options  are  supported  by  an  implementation  of  a given  standard.  See  PICS  proforma. 

Protocol  Implementation  eXtra  Information  for  Testing  (PIXIT):  a statement  made  by  a 
supplier  who  desires  testing  laboratory  services  which  contains  information  about  the  lUT 
and  its  computer  environment  that  is  needed  by  the  testing  laboratory  to  test  the  lUT. 

System  Under  Test  (SUT):  the  computer  environment  that  includes  the  hardware,  software 
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and  communication  network  required  to  support  the  lUT. 

test  campaign:  the  process  of  running  an  executable  version  of  the  abstract  test  suite  against 
a particular  lUT  and  capturing  the  test  results. 

(abstract)  test  method:  the  description  of  how  an  implementation  is  to  be  tested  at  an 
appropriate  level  of  abstraction  to  make  this  description  independent  of  any  particular 
implementation  of  testing  tools  or  procedures. 

test  purpose:  a precise  description  of  an  objective  which  an  abstract  test  case  is  designed  to 
evaluate. 

(conformance)  test  report  : a document  written  at  the  end  of  the  conformance  assessment 
process,  that  provides  the  overall  summary  of  the  conformance  of  the  lUT  to  the  standard  for 
which  conformance  testing  was  carried  out,  and  that  gives  the  details  of  the  testing. 

testing  laboratory:  an  organization  that  carries  out  the  conformance  assessment  process.^ 

(test)  verdict:  a statement  assigned  from  the  analysis  of  the  test  results  from  the  execution  of 
a test  case  by  an  lUT. 

verdict  criteria:  information  on  the  allowable  states  of  conditions  or  values  of  variables  that 
must  exist  in  a test  result  to  satisfy  a program  state  defined  within  an  abstract  test  case, 
enabling  the  assignment  of  a verdict  to  a particular  test  result. 

3.  Background 

ISO  10303-203  is  the  reference  standard  for  this  beta  testing  program.  This  specification 
uses  elements  of  other  parts  of  ISO  10303  and  other  standards  as  appropriate.  An  overview 
of  ISO  10303  is  provided  in  ISO  10303-1  which  describes  the  relationships  among  the 
documents  which  make  up  the  standard."^  The  list  of  references  to  these  and  other  documents 
used  by  this  beta  testing  program,  along  with  their  specific  version  is  provided  in  Appendix 
A. 

The  draft  international  standard  version  (DIS)  of  ISO  10303-203  will  be  used  initially.  The 
testing  will  be  upgraded  to  account  for  the  minor  differences  in  the  international  standard  by 
early  1995. 


^ A testing  laboratory  can  be  a third-party  testing  laboratory,  a user  organization,  or  an 
independent  unit  of  a supplier  organization. 

A full  description  of  the  series  of  documents  within  ISO  10303  is  contained  in  ISO 
10303-1,  Overview  and  Fundamental  Principles. 
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ISO  10303-203  is  one  of  a series  of  documents  called  Application  Protocols  (AP).  AP 
specifications  (ISO  10303  200-series  documents)  were  designed  to  be  implemented  to  support 
end-user  applications.  APs  specify  the  information  requirements  for  data  exchange.  An  AP 
specifies  a data  representation  which  is  used  to  satisfy  these  requirements.  Conformance 
requirements  are  also  defined  and  these  requirements  may  be  grouped  into  conformance 
classes.  A conforming  implementation  may  not  arbitrarily  support  some  of  the  requirements. 
It  must  support  all  requirements  within  a conformance  class.  A companion  document  to  each 
AP  is  the  abstract  test  suite  (ATS)  (ISO  10303  1200-series  documents).  An  ATS  states  the 
purposes  for  tests  and  the  verdict  criteria  applied  to  each  test  result  to  determine  if  the 
implementation  passed  the  test.  Since  ISO  10303  uses  a formal  data  specification  language, 
called  EXPRESS  (ISO  10303-11),  most  test  purposes  may  be  automatically  derived. 

An  AP  uses  at  least  one  implementation  method  (ISO  10303  20-series  document)  which 
specifies  the  requirements  for  formatting  the  data  representation  for  exchange  or  data 
sharing.  The  beta  testing  program  supports  only  the  first  of  these.  Implementation  Methods: 
Clear  Text  Encoding  of  the  Exchange  Structures  (ISO  10303-21).  Support  for  other 
implementation  methods  will  be  added  as  they  become  part  of  the  international  standard.^ 

A supplier  of  an  implementation  may  claim  conformance  to  one  or  more  of  the  portions  of 
AP  203  as  defined  by  conformance  classes.  The  supplier  of  an  implementation  desiring 
testing  services  is  requested  to  complete  a statement  that  identifies  which  conformance 
classes  are  supported  by  their  implementation,  called  a protocol  implementation  conformance 
statement  (PICS).  This  statement  is  used  to  select  which  tests  cases  from  the  abstract  test 
suite  (ATS)  are  appropriate  to  use  in  determining  a claim  of  conformance.  An  implementor 
may  also  use  this  statement  to  determine  which  test  cases  should  be  executed  during  self- 
testing. 

The  requirements  for  the  conformance  testing  of  implementations  are  specified  in  ISO  10303- 
31,  Conformance  Testing  Methodology  and  Framework:  General  Concepts.  Conformance 
testing  is  conducted  via  national  testing  programs.  This  beta  testing  program  is  being 
conducted  to  prepare  for  the  creation  of  a national  testing  program  for  U.S.  industry  and 
government.  The  use  of  the  testing  program  by  industry  is  voluntary.  Experience  from 
other  standards  efforts  has  identified  the  importance  of  establishing  a testing  program  that  is 
timed  to  support  the  emergence  of  commercial  products.  The  quality  of  the  initial  set  of 
commercial  products  can  significantly  impact  the  rate  at  which  a standard  is  adopted.  The 
existence  of  a national  testing  program  may  also  offer  some  competitive  advantage  to  vendors 
of  commercial  products. 


^ For  example,  ISO  10303-22,  Implementation  Methods:  Standard  Data  Access  Interface, 
is  an  application  program  interface  specification  which  has  completed  its  first  international 
ballot. 
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4.  Testing  Program  Policy 

The  STEP  beta  testing  program  will  be  used  to  achieve  an  ac:  :ptable  degree  of  confidence  in 
the  test  procedures,  the  conformance  testing  system,  and  the  test  methods.  Together,  these 
components  provide  a means  of  testing  an  implementation  for  conformance  to  a standard. 
Confidence  in  these  components  must  be  attained  before  a recognized  testing  program  can 
commence  and  certificates  of  conformity  can  be  issued. 

To  obtain  the  feedback  necessary  to  obtain  this  confidence,  testing  will  be  conducted  by 
ITI/CEC  and  NIST/MEL  for  the  duration  of  the  beta  testing  program.  ITI/CEC  is 
coordinating  the  beta  conformance  testing  program.  All  functions  relating  to  the  audit  of  test 
results  will  be  conducted  by  NIST/MEL.  NIST/MEL  and  ITI/CEC  reserve  the  right  to  limit 
the  number  of  systems  evaluated  during  the  beta  testing  program. 

Until  a recognized  testing  program  is  initiated,  commercial  systems  which  have  successfully 
completed  the  conformance  assessment  process  will  be  recognized  in  an  appropriate 
publication  in  place  of  receiving  a certificate  of  conformance. 

The  beta  testing  program  will  continue  until  confidence  in  the  test  system,  test  suite  and  test 
procedures  has  been  achieved.  There  is  no  fixed  duration  for  the  beta  testing  program  but 
NIST  and  ITI  have  estimated  the  duration  of  the  program  to  be  about  one  year.  NIST  plans 
to  create  a national  STEP  testing  program  for  government  users  but  this  testing  program  may 
also  be  used  by  industry.  The  authority  for  establishing  such  services  was  provided  under 
the  Federal  Property  and  Administrative  Services  Act  of  1949,  as  amended  by  Public  Law 
100-235.  A Federal  Information  Processing  Standard  (FIPS)  which  aids  in  procuring 
conforming  systems  is  being  pursued  for  ISO  10303.  NIST  plans  to  initiate  a STEP 
conformance  testing  service  and  the  FIPS  will  require  conformance  testing.  After  the  initial 
start  up  costs,  the  testing  program  will  be  funded  from  user  fees. 

The  intent  is  to  transition  the  use  of  testing  laboratories  which  have  been  accredited  through 
the  National  Voluntary  Laboratory  Accreditation  Program  (NVLAP).  NIST/MEL  will  be 
responsible  for  the  review  of  test  reports  and  the  issuance  of  certificates  of  conformity. 
Certificates  will  be  issued  only  after  the  successful  completion  of  the  conformance  assessment 
process  as  defined  in  ISO  10303-31. 

4.1  Client  Confidentiality 

As  with  other  testing  programs  of  this  nature,  any  information  relating  to  a supplier  (e.g.  the 
client  for  testing  services)  whose  system  is  under  test  remains  confidential  until  testing  is 
completed  and  the  client  agrees  to  the  release  of  test  results. 

The  remainder  of  this  section  provides  the  planned  process  and  policies  for  the  national 
testing  program  for  STEP  (as  opposed  to  the  less  formal  beta  testing  program).  Section  5 
and  section  6 contain  information  and  policies  specific  to  the  beta  testing  program. 
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4.2  Organizational  Model  and  Conformance  Testing  Process  Overview 

There  are  two  perspectives  that  must  be  supported  by  a conformance  testing  policy.  The 
policy  must  support  both  the  conducting  of  the  testing  and  the  evaluation  of  the  capabilities 
of  the  organization  performing  the  testing. 

Accreditation  is  the  administrative  activity  of  recognizing  a testing  laboratory  is  qualified  to 
conduct  conformance  testing. 

Certification  is  the  administrative  activity  of  recognizing  a system  has  demonstrated 
conformance  to  a standard,  and  of  publicly  registering  the  test  results. 

The  organizational  model  consists  of  clients  who  seek  certification  of  their  software  product, 
testing  laboratories  who  set  up  the  testing  and  generate  the  test  reports,  and  a certification 
authority  who  issues  certificates  after  an  administrative  review.  The  organizational  model  is 
consistent  with  the  model  described  in  ISO  10303-32,  Conformance  Testing:  Requirements 
for  Testing  Laboratories  and  Clients,  and  the  model  used  in  other  NIST  Testing  Programs.^ 

Conformance  Testing  Process 

1.  Client  contacts  NVLAP  or  NIST/MEL  for  list  of  accredited  Testing  Laboratories.^ 

2.  Client  contacts  an  accredited  Testing  Laboratory  and  states  services  needed. 

3.  Client  and  Testing  Laboratory  agree  on  services  and  a contract  is  written  to  specify  fees 
and  services. 

4.  Client  provides  product  to  be  tested  and  supporting  documentation. 

5.  Testing  Laboratory  provides  testing  services  and  expertise  as  needed. 


^ The  NIST  model  is  described  in  a public  notice  titled  "Conformance  Testing  Policy  and 
Procedures"  in  the  Federal  Register  dated  August  3,  1988. 

^ The  National  Voluntary  Laboratory  Accreditation  Program  produces  a quarterly  report 
that  lists  accredited  testing  laboratories.  Copies  can  be  obtained  from  NIST,  NVLAP  Office, 
Building  TFR  Room  A 162,  Gaithersburg,  MD  20899,  telephone: (301)  975-4017. 

NIST/MEL  maintains  the  STEP  On-line  Information  System  (SOLIS)  which  contains 
documents  and  current  information  on  ISO  10303.  When  testing  laboratories  for  ISO  10303 
are  established  and  accredited,  an  area  will  be  created  on  SOLIS  that  contains  a listing  of 
these  laboratories. 
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6.  If  a certificate  of  conformance  is  the  objective  of  testing,  the  Testing  Laboratory  forwards 
the  test  results  and  supporting  documentation  to  NIST  for  evaluation. 

7.  NIST  evaluates  the  test  results  and,  if  acceptable,  issues  a certificate  of  conformance. 

4.3  NIST  STEP  Certification  Policy 

NIST  will  provide  the  overall  direction  for  organizing  managing,  and  administering  the 
STEP  conformance  testing  and  certification  program.  NIST  responsibilities  in  this  capacity 
will  include: 

a)  establishing  and  maintaining  the  conformance  testing  program  policies  and 
procedures; 

b)  approving  the  test  methods  used  to  determine  the  compliance  of  products; 

c)  insuring  that  the  test  methods  are  maintained; 

d)  evaluating  and  resolving  disputes  on  all  matters  concerning  conformance  testing; 

e)  establishing,  with  the  National  Voluntary  Laboratory  Accreditation  Program 
(NVLAP),  the  accreditation  criteria  for  testing  laboratories; 

f)  maintaining  and  publishing  a list  of  recognized  testing  laboratories; 

g)  establishing  the  fees  or  rates  for  NIST  provided  services  or  products; 

h)  coordinating  with  other  certification  authorities  to  review  certification  criteria  for 
the  purpose  of  harmonizing  procedures  and  fostering  mutual  recognition  of 
certification; 

i)  developing  and  maintaining  procedures  for  clients  to  follow  in  order  to  receive  a 
Certificate  of  Conformity; 

j)  issuing  the  Certificate  of  Conformity  based  on  the  satisfactory  evaluation  of  test 
reports  and  supporting  documentation; 

k)  maintaining  and  publishing  a register  of  products  on  the  Validated  Products  List 
that  have  received  a Certificate  of  Conformity;  and 

l)  periodically  assessing  the  need  for  a conformance  testing  program  and  maintenance 
of  test  methods. 

4.4  NIST  STEP  Accreditation  Policy  for  Testing  Laboratories 
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NIST  will  carry  out  its  responsibilities  for  conformance  certification  through  testing 
laboratories  judged  to  be  competent  and  objective.  NIST  will  draw  on  the  National  Voluntary 
Laboratory  Accreditation  Program  (NVLAP)  for  accrediting  testing  laboratories.  The 
process  and  requirements  are  described  in  the  NVLAP  Procedures.  Testing  Laboratory 
responsibilities  include: 

a)  obtaining  and  maintaining  laboratory  accreditation; 

b)  conducting  conformance  testing  in  accordance  with  the  current  test  methods,  using 
approved  tools  and  procedures; 

c)  preparing  test  results  in  accordance  with  the  requirements  specified  in  ISO  10303- 
32,  Conformance  Testing  - Requirements  for  Testing  Laboratory  and  Clients,  and 
specified  in  the  NIST  testing  policy  on  certificates  of  conformity; 

d)  providing  feedback  on  problems  and  improvements  to  the  test  suites,  test  methods, 
test  system,  and  conformance  testing  procedures; 

e)  conforming  to  the  guidelines  for  testing  laboratories  provided  in  the  NVLAP 
Program  Handbook', 

f)  treating  all  test  results  confidentially; 

g)  paying  all  relevant  fees  and  establishing  fees  for  Laboratory  provided  services  or 
products;  and 

h)  participating  in  training  sessions  and  reviewing  material  as  required  to  address 
changes  to  testing  methods  and  conformance  testing  procedures. 

Testing  Laboratories  have  the  right  to  publish  and  advertise  their  accredited  status  as  long  as 
the  advertising  does  not  imply  product  certification  by  NIST  or  the  U.S.  Government. 

4.5  Disputed  Test  Results 

As  with  other  testing  programs  of  this  nature,  a Test  Method  Control  Board  will  be  formed 
to  arbitrate  any  disputes  arising  from  the  performance  of  testing  in  accordance  with  the 
planned  FIPS  and  the  approved  test  method.  It  will  be  the  responsibility  of  NIST  to  insure 
that  the  test  method  is  properly  maintained  and  that  any  changes  are  properly  coordinated. 

The  Control  Board  procedures  will  include  provisions  for: 

a)  handling  disputes  over  the  validity  of  the  test  method; 

b)  approving  changes  to  the  test  method; 
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c)  validating  and  releasing  new  versions  of  the  test  method; 

d)  coordinating  any  problems  associated  with  the  STEP  standard  back  into  the 
appropriate  standard’s  project  and  any  changes  to  the  test  methods  to  accredited 
Testing  Laboratories;  and 

e)  assisting  Testing  Laboratories  and  Clients  in  the  proper  use  of  the  test  method. 

4.6  Suimnary  of  Proposed  Testing  Program  Implementation  and  Administration 
The  STEP  testing  program,  when  fully  operational,  will  have  the  following  characteristics: 

• testing  will  be  conducted  by  accredited  Testing  Laboratories; 

• NIST  will  review  test  results  and  issue  certificates  of  conformance.  (A  fee  to  cover 
administrative  costs  will  be  assessed); 

• the  test  suite  and  other  supporting  documentation  (see  5.3  for  the  list  for  the  beta 
testing  program)  will  be  available  for  a fee  from  NIST  or  an  organization  providing 
this  service  for  NIST; 

• NIST  will  distribute  "fixes"  to  the  test  suite  and  "interpretations"  to  the  testing  policy 
or  procedures  as  required. 

5.  Beta  Testing  Program 

5.1  Availability 

The  beta  testing  program  for  AP  203  is  designed  to  meet  the  general  requirements  defined  in 
ISO  10303-31,  Conformance  Testing  Methodology  and  Framework:  General  Concepts.  The 
program  evaluates  implementations  for  conformance  to  ISO/DIS  10303-203,  Application 
Protocol:  Configuration  Controlled  3D  Design  for  Mechanical  Parts  and  Assemblies,  based 
on  the  test  methods  specified  in  ISO  10303-34,  Conformance  Testing  Methodology  and 
Framework:  Abstract  Test  Methods. 

The  STEP  Conformance  Test  System  is  designed  to  test  conformance  to  any  STEP  AP  . The 
beta  AP  203  Conformance  Test  Suite  specifically  tests  the  assertions  applicable  to  ISO/DIS 
10303-203.  The  test  suite  for  the  version  of  ISO  10303-203  that  is  an  international  standard 
should  be  available  in  early  1995.  The  released  version  of  the  test  suite  covers  the  majority 
of  the  requirements  in  ISO  10303-203  and  efforts  are  continuing  to  increase  the  coverage  of 
this  test  suite. 

The  beta  testing  program  will  provide  the  STEP  Conformance  Test  System,  the  AP  203  Test 
Suite,  documentation  and  the  testing  policy  and  procedures  (see  Appendix  A).  The  test 
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system  interface  is  designed  to  operate  in  a computer-aided  software  engineering  (CASE) 
environment  to  facilitate  its  use  for  self-testing  by  vendors.  Updates  to  the  test  suite  or 
procedures  will  also  be  provided,  as  they  occur,  via  electronic  distribution.  Implementors 
who  successfully  complete  self-testing  should  be  confident  in  their  ability  to  pass 
conformance  testing  with  an  independent  test  laboratory.  Implementors  who  wish  to 
participate  in  the  beta  testing  program  should  send  a request  to: 

Robert  Matthews 
Industrial  Technology  Institute 
Hubbard  Drive 
Ann  Arbor,  MI 
e-mail:  rsm@iti.com 
(313)  769-4573 

ITI/CEC  or  NIST/MEL  will  contact  the  requesting  organization  to  arrange  for  a mutually 
agreeable  time  to  conduct  this  testing. 

5.2  Implementor  Collaboration 

In  an  effort  to  maximize  the  effectiveness  of  AP  203  beta  testing,  ITI  and  NIST  will  provide 
to  participants  in  the  program  the  following: 

- the  STEP  Beta  Conformance  Testing  Kit  and  testing  procedures; 

- updates  to  test  suite  and  test  procedures  as  they  occur; 

- published  announcement  of  the  lUT’s  successful  completion  of  ISO  DIS  10303-203 
beta  testing;  and 

- highest  priority  consideration  for  conformance  testing  when  the  formal  AP  203 
service  is  up  and  running. 

In  return,  the  implementor  is  requested  to  provide: 

- copies  of  the  reports  generated  under  5.4  and  6.3  below;  and 

- feedback  on  any  test  suite  or  test  procedure  deficiencies. 

5.3  Testing  Resources 

The  following  resources  are  available  for  performing  conformance  testing  of  AP  203: 

• STEP  Beta  Conformance  Testing  Kit  which  includes: 

- a "READ-ME"  sheet  explaining  what  is  in  the  kit,  and  why  it  is  there; 

- registration  forms  for  scheduling  testing  under  the  "beta"  program; 

“ a list  of  points  of  contact; 

- a pre-test  preparation  checklist  and  a description  on  how  to  prepare  for  testing; 

- the  AP  203  Test  Suite  (distributed  on  electronic  media); 

- sample  reports  (See  5.4)  with  an  explanation  on  what  each  means; 

- a sample  of  a test  case  with  a description  of  how  the  testing  is  conducted; 
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• the  system  to  be  tested,  called  the  Implementation  Under  Test  (lUT); 

• sufficient  disk  space  to  load  the  AP  203  Test  Suite; 

• the  ability  to  input  and  generate  STEP  exchange  structures; 

• an  operator  who  is  proficient  with  the  lUT  and  the  environment  in  which  the  lUT 
runs  (he/she  will  be  required  to  follow  a set  of  instructions  for  entering  data  into  the 
lUT  and  answer  a set  of  questions  about  the  lUT  treatment  of  this  data);  and 

• adequate  time  to  install  and  execute  the  test  suite. ^ 

If  the  lUT  is  receiving  testing  services  at  one  of  the  beta  testing  laboratories,  the  STEP 
Conformance  Test  System  is  not  required.  An  implementor  will  need  the  following,  in 
addition  to  the  items  list  above,  for  the  STEP  Conformance  Test  System: 

• the  STEP  Conformance  Test  System  software  (distributed  on  electronic  media),  and 
its  supporting  documentation; 

• the  instructions  on  how  to  interface  the  system  to  be  tested  to  the  STEP 
Conformance  Test  System;  interface  requirements  and  options  are  also  defined;  and 

• sufficient  disk  space  to  load  compile  and  run  the  STEP  Conformance  Test  System. 
5.4  Test  Reports 

Two  reports  are  to  be  generated  by  the  organization  requesting  testing  services.  These  are: 

• a PICS  which  is  generated  using  the  Proforma  PICS;  and 

• aPIXIT. 

Three  reports  are  generated  during  the  conformity  assessment  process.  These  are: 

• Installation  Report:  Captures  any  commands  used  to  install  and  configure  the  test 
system; 


^ For  estimation  purposes,  installation  typically  requires  less  than  3 hours  and  the  test 
suite  is  designed  to  be  executed  in  8 hours  or  less.  These  estimates  may  vary  significantly 
depending  upon  the  computer  hardware  and  system  under  test. 
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• Conformance  Log:  Provides  a step-by-step  record  of  what  tests  were  performed, 
what  preparations  were  taken,  and  what  result  was  achieved  for  each  test;  and 

• (Conformance)  Test  Report:  Provides  the  identification  of  each  test  executed,  the 
status  of  each  test,  and  the  verdict  assigned  to  each  test  result. 

The  installation  report  and  conformance  log  provide  supporting  documentation  that  may  be 
used  to  isolate  the  conditions  of  a particular  test  or  test  campaign. 

6.  Testing  Procedures 

6.1  Requirements 

The  current  AP  203  Conformance  Test  Suite  shall  be  used  whenever  conformance  testing  for 
AP  203  is  performed.  The  most  recent  release  of  the  test  suite  is  to  be  used  in  conjunction 
with  any  enhancements  that  are  available.  An  implementor  may  utilize  the  test  suite  as 
he/she  sees  fit,  modifying  or  enhancing  it  to  provide  added  value.  However,  when  a test  for 
conformance  is  to  be  performed,  the  UNEDITED  version  of  the  test  suite  must  be  used. 

The  STEP  Conformance  Test  System  was  designed  to  also  be  used  during  the  development  of 
the  implementation. 

Documentation  of  the  system  environment  is  required  and  a questionnaire  is  provided  which 
asks  for  relevant  information  on  the  hardware  and  software  configuration.  A combination  of 
software  components  may  be  used  together  to  form  the  lUT.  If  these  components  are 
distinguishable  and  each  has  its  own  software  release  revision  identification,  a list  of  these 
components  and  their  release  identification  are  required. 

6.2  Establishiug  a Test  Euviroumeut 

The  User  Manual  for  the  STEP  Conformance  Test  System  provides  the  documentation 
required  to  install,  configure  and  execute  the  AP  203  Conformance  Test  Suite.  It  is  the 
responsibility  of  the  implementor  to  configure  the  lUT  and  the  environment  in  which  it 
operates  so  adequate  information  can  be  captured  during  testing  to  assign  a verdict. 

In  the  beta  release  of  the  STEP  Conformance  Test  System  and  the  AP  203  Test  Suite,  some 
operations  are  performed  manually  by  the  operator.  Based  on  observations  recorded,  the  test 
system  is  used  to  assign  a verdict  to  each  test  case.  An  implementation  must  pass  all  test 
cases  associated  with  a conformance  class  or  option  for  which  a claim  of  conformance  is 
made  to  be  recognized  as  successfully  completing  the  conformance  assessment  process. 

If  an  error  is  determined  in  the  configuration,  the  AP  203  Conformance  Test  Suite  must  be 
rerun  in  its  entirety. 
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6.3  Documentation  Requirements 

When  testing  services  are  desired,  a documentation  review  is  performed  by  the  test 
laboratory  (for  the  beta  testing  program  NIST/MEL  or  ITI/CEC)  of  the  PICS  and  PIXIT  to 
determine  their  completeness  prior  to  agreeing  to  initiating  a test  campaign^.  The  PICS 
identifies  if  a claim  of  conformance  is  being  made  for  each  conformance  class.  Each  test 
case  in  the  test  suite  identifies  the  conformance  class  to  which  it  applies.  This  identification 
is  used  to  select  which  tests  are  appropriate  to  execute. 

6.4  Test  Reports 

The  Installation  Report  is  completed  by  the  operator  of  the  system  under  test.  It  should  be 
given  a filename  of  im tall. log. 

The  Conformance  Log  provides  sufficient  detail  to  isolate  failures.  An  entry  is  produced  for 
each  element  tested.  Results  are  stored  in  a filename  of  campaign.log. 

The  Conformance  Test  Report  contains  a verdict  assigned  to  a test  result  of  each  test 
executed  based  on  pre-established  verdict  criteria.  The  possible  verdict  assignments  are: 

pass  — successful  test  of  an  assertion; 

fail  “ unsuccessful  test  of  an  assertion; 

inconclusive  - verdict  could  not  be  assigned  due  to  failure  of  a related  test  or 

error  in  the  test  suite; 

untested  - could  not  exercise  the  test  assertion. 

6.5  Recognition  of  Conformance  to  ISO  10303-203 

During  the  beta  conformance  testing  program,  NIST  will  announce  products  which  have 
successfully  completed  conformance  assessment  through  the  US  Pro  Exchange  newsletter 
under  the  article  title  Register  of  Tested  STEP  Products.  NIST  will  also  maintain  an  on-line 
listing  of  these  products  on  SOLIS. 

Certificates  of  conformance  will  only  be  issued  by  NIST  once  the  STEP  Conformance 
Testin.  Service  is  established.  Successfully  tested  products  will  be  listed  in  the  Validated 
Products  List. 


^ For  self-testing,  NIST  would  like  the  PICS  and  PIXIT  to  be  included  with  the  test 
results. 
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7.  Summary 

NIST  plans  to  provide  the  overall  direction  for  organizing  managing  and  administering  a 
national  STEP  conformance  testing  and  certification  program.  The  testing  program  will  be 
available  to  government  agencies  and  industry.  NIST  will  issue  certificates  of  conformity  to 
suppliers  who  meet  the  requirements  of  the  testing  program.  This  program  is  being 
undertaken  to  encourage  the  availability  of  high-quality  implementations  of  STEP,  thereby 
accelerating  the  broad  adoption  of  the  standard  by  industry.  The  STEP  conformance  testing 
program  may  be  used  by  both  government  and  industry  but  the  use  of  the  testing  program  by 
industry  is  voluntary. 

A beta  testing  program  is  being  initiated  to  develop  confidence  in  the  test  methods  and 
procedures.  The  beta  testing  program  will  accomplish  this  through  the  trial  use  of  these  test 
methods  and  procedures  with  suppliers  of  early  implementations  of  ISO  10303-203.  Strong 
vendor  involvement  is  needed  to  obtain  feedback  on  all  aspects  of  the  proposed  test  program 
to  ensure  value-added  testing  services. 
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Appendix  A Sources  of  Documents 


US  PRO,  care  of  National  Computer  Graphics  Association,  P.O.  Box  #S,  2722  Merrilee 
Drive,  Suite  200,  Merrifield  VA,  telephone:  (703)698-9600,  ext.  325,  FAX  (703)  560-2752. 

• ISO  10303-1:1994,  Industrial  automation  systems  and  integration  — Product 
data  representation  and  exchange  — Part  1:  Overview  and  fundamental 
principles. 

• ISO  10303-21:1994,  Industrial  automation  systems  and  integration  — Product 
data  representation  and  exchange  — Part  21  Implementation  methods:  Clear 
text  encoding  of  the  exchange  structure;. 

• ISO  10303-31:1994,  Industrial  automation  systems  and  integration  — Product 
data  representation  and  exchange  — Part  31  Conformance  testing  methodology 
and  framework:  General  concepts;. 

• ISO/DIS  10303-203:1993,  Industrial  automation  systems  and  integration  — 
Product  data  representation  and  exchange  — Part  203  Application 
Protocol:  Configuration  Controlled  3D  Design  for  Mechanical  Parts  and 
Assemblies 

• VS  PRO  Exchange  Newsletter 

National  Institute  of  Standards  and  Technology  (NIST),  Manufacturing  Engineering 
Laboratory,  Building  220  Room  A127,  Gaithersburg,  MD  20899,  telephone: (301)  975-3983, 
FAX  (301)  258-9749. 

• ISO/CD  10303-32:1994,  Industrial  automation  systems  and  integration  — 
Product  data  representation  and  exchange  —Conformance  Testing: 
Requirements  for  Testing  Laboratories  and  Clients 

• ISO/WD  10303-1203:1994,  Industrial  automation  systems  and  integration  — 
Product  data  representation  and  exchange  — Part  1203  Abstract  Test 
Suite  .’Configuration  Controlled  3D  Design  for  Mechanical  Parts  and 
Assemblies 

• Initial  NIST  Testing  Policy  for  STEP  - Requirements  for  Conformance 
Assessment  of  AP  203  Implementations 

• Issues,  Requirements,  and  Recommendations  for  a STEP  Conformance  Testing 
Program,  NISTIR  4743,  January  1993. 

• STEP  On-line  Information  System  (SOLIS)  NISTIR  pending,  October  1994. 
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Industrial  Technology  Institute  (ITI),  Center  for  Electronic  Commerce,  P.O.  Box  1485,2901 
Hubbard  Road,  Ann  Arbor,  MI  48106,  telephone:  (313)  769-4000,  FAX  (313)  769-4064. 

• STEP  Beta  Conformance  Test  Kit,  September  1994  (This  contains  the 

instructions,  documents  and  software  described  in  5.3). 

Other  references 

National  Institute  of  Standards  and  Technology  (NIST),  Laboratory  Accreditation  Program, 
Building  TFR  Room  A 162,  Gaithersburg,  MD  20899,  telephone: (301)  975-4017. 

• NVLAP  Procedures,  Special  Publication  150 

National  Technical  Information  Sservice  (NTIS),  U.  S.  Department  of  Commerce,  5285 
Port  Royal  Road,  VA  22151,  telephone:  subscriptions  (703)  487-4630,  individual  issues 
(703)  487-4650. 

• Validated  Products  List,  Judy  Kailey,  editor.  National  Institute  of  Standards  and 
Technology  (NIST),  NISTIR  5510,  October  1994.  Published  quarterly.  Ordering 
Number:  PB93-937303/AS.  Also  available  via  the  Internet  using  ftp  address 
speckle.ncsl.nist.gov  or  internet  address  129.6.59.2. 
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